<!DOCTYPE html>
<html>
<head>
    

    

    



    <meta charset="utf-8">
    
    
    <link rel="canonical" href="http://xiejm.com/Hadoop/Hadoop--Fully_Distributed _Mode_HA.html">
    
    
    <title>Hadoop--集群(Fully Distributed Mode)模式部署+HA | XieJM&#39;s Blog | 建立博客是为了记录工作经验以及生活点滴,也是将知识和经验分享给需要的朋友，希望对你有帮助！</title>
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
    
    <meta name="theme-color" content="#00bcd4">
    
    
    <meta name="keywords" content="Hadoop">
    <meta name="description" content="本文主要记录Hadoop全分布模式（Fully Distributed Mode）部署的详细步骤： Hadoop全分布模式（Fully Distributed Mode）是一种集群部署方式，它的守护进程运行在一个集群上。">
<meta name="keywords" content="Hadoop">
<meta property="og:type" content="article">
<meta property="og:title" content="Hadoop--集群(Fully Distributed Mode)模式部署+HA">
<meta property="og:url" content="http://xiejm.com/Hadoop/Hadoop--Fully_Distributed _Mode_HA.html">
<meta property="og:site_name" content="XieJM&#39;s Blog">
<meta property="og:description" content="本文主要记录Hadoop全分布模式（Fully Distributed Mode）部署的详细步骤： Hadoop全分布模式（Fully Distributed Mode）是一种集群部署方式，它的守护进程运行在一个集群上。">
<meta property="og:locale" content="zh-CN">
<meta property="og:updated_time" content="2017-10-03T13:06:39.437Z">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Hadoop--集群(Fully Distributed Mode)模式部署+HA">
<meta name="twitter:description" content="本文主要记录Hadoop全分布模式（Fully Distributed Mode）部署的详细步骤： Hadoop全分布模式（Fully Distributed Mode）是一种集群部署方式，它的守护进程运行在一个集群上。">
    
        <link rel="alternate" type="application/atom+xml" title="XieJM&#39;s Blog" href="/atom.xml">
    
    <link rel="shortcut icon" href="/favicon.ico">
    <link rel="stylesheet" href="/css/style.css?v=1.6.13">
    <script>window.lazyScripts=[]</script>

    <!-- custom head -->
    

</head>

<body>
    <div id="loading" class="active"></div>

    <aside id="menu" class="hide" >
  <div class="inner flex-row-vertical">
    <a href="javascript:;" class="header-icon waves-effect waves-circle waves-light" id="menu-off">
        <i class="icon icon-lg icon-close"></i>
    </a>
    <div class="brand-wrap" style="background-image:url(/img/brand.jpg)">
      <div class="brand">
        <a href="/" class="avatar waves-effect waves-circle waves-light">
          <img src="/img/avatar.jpg">
        </a>
        <hgroup class="introduce">
          <h5 class="nickname">XieJM</h5>
          <a href="mailto:309469843@qq.com" title="309469843@qq.com" class="mail">309469843@qq.com</a>
        </hgroup>
      </div>
    </div>
    <div class="scroll-wrap flex-col">
      <ul class="nav">
        
            <li class="waves-block waves-effect">
              <a href="/index.html"  >
                <i class="icon icon-lg icon-home"></i>
                主页
              </a>
            </li>
        
            <li class="waves-block waves-effect">
              <a href="/archives/index.html"  >
                <i class="icon icon-lg icon-archives"></i>
                归档
              </a>
            </li>
        
            <li class="waves-block waves-effect">
              <a href="/tags/index.html"  >
                <i class="icon icon-lg icon-tags"></i>
                标签
              </a>
            </li>
        
            <li class="waves-block waves-effect">
              <a href="/categories/index.html"  >
                <i class="icon icon-lg icon-th-list"></i>
                分类
              </a>
            </li>
        
            <li class="waves-block waves-effect">
              <a href="https://github.com/xjmhz" target="_blank" >
                <i class="icon icon-lg icon-github"></i>
                Github
              </a>
            </li>
        
      </ul>      
    </div>
    <footer class="footer">
    <p>欢迎加入我们的大数据交流群：<br>群1：258669058 群2：126181630</p>   
    <p>        
        <span><a rel="license" href="https://creativecommons.org/licenses/by-nc-sa/4.0/deed.zh"><img src="/img/cc.png"></a></span>
        
        <span><a href="/atom.xml" target="_blank" class="rss" title="rss"><i class="icon icon-2x icon-rss-square"></i></a></span>
        
    </p>
    <p><span>XieJM &copy; 2017</span>
    </p>
    <p><span>
            
            Power by <a href="http://hexo.io/" target="_blank">Hexo</a> Theme <a href="https://github.com/yscoder/hexo-theme-indigo" target="_blank">indigo</a>
        </span>
    </p>
</footer>
  </div>
</aside>

    <main id="main">
        <header class="top-header" id="header">
    <div class="flex-row">
        <a href="javascript:;" class="header-icon waves-effect waves-circle waves-light on" id="menu-toggle">
          <i class="icon icon-lg icon-navicon"></i>
        </a>
        <div class="flex-col header-title ellipsis">Hadoop--集群(Fully Distributed Mode)模式部署+HA</div>
        
        <div class="search-wrap" id="search-wrap">
            <a href="javascript:;" class="header-icon waves-effect waves-circle waves-light" id="back">
                <i class="icon icon-lg icon-chevron-left"></i>
            </a>
            <input type="text" id="key" class="search-input" autocomplete="off" placeholder="输入感兴趣的关键字">
            <a href="javascript:;" class="header-icon waves-effect waves-circle waves-light" id="search">
                <i class="icon icon-lg icon-search"></i>
            </a>
        </div>
        
        
        <a href="javascript:;" class="header-icon waves-effect waves-circle waves-light" id="menuShare">
            <i class="icon icon-lg icon-share-alt"></i>
        </a>
        
    </div>
</header>
<header class="content-header post-header">

    <div class="container fade-scale">
        <h1 class="title">Hadoop--集群(Fully Distributed Mode)模式部署+HA</h1>
        <h5 class="subtitle">
            
                <time datetime="2017-10-03T12:33:09.754Z" itemprop="datePublished" class="page-time">
  2017-10-03
</time>


	<ul class="article-category-list"><li class="article-category-list-item"><a class="article-category-list-link" href="/categories/tech/">技术</a></li></ul>

            
        </h5>
    </div>

    


</header>


<div class="container body-wrap">
    
    <aside class="post-widget">
        <nav class="post-toc-wrap" id="post-toc">
            <h4>TOC</h4>
            <ol class="post-toc"><li class="post-toc-item post-toc-level-2"><a class="post-toc-link" href="#1-基本环境"><span class="post-toc-text">1.基本环境</span></a><ol class="post-toc-child"><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#1-1组件说明："><span class="post-toc-text">1.1组件说明：</span></a></li></ol></li><li class="post-toc-item post-toc-level-2"><a class="post-toc-link" href="#2-主机节点规划"><span class="post-toc-text">2.主机节点规划</span></a></li><li class="post-toc-item post-toc-level-2"><a class="post-toc-link" href="#3-目录规划"><span class="post-toc-text">3.目录规划</span></a></li><li class="post-toc-item post-toc-level-2"><a class="post-toc-link" href="#4-系统环境配置"><span class="post-toc-text">4.系统环境配置</span></a><ol class="post-toc-child"><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#4-1-配置网络和主机名"><span class="post-toc-text">4.1 配置网络和主机名</span></a><ol class="post-toc-child"><li class="post-toc-item post-toc-level-4"><a class="post-toc-link" href="#网络IP设置"><span class="post-toc-text">网络IP设置</span></a></li><li class="post-toc-item post-toc-level-4"><a class="post-toc-link" href="#主机名配置"><span class="post-toc-text">主机名配置</span></a></li></ol></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#4-2-添加集群用户"><span class="post-toc-text">4.2 添加集群用户</span></a></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#4-3-关闭SELINUX"><span class="post-toc-text">4.3 关闭SELINUX</span></a></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#4-4-关闭防火墙"><span class="post-toc-text">4.4 关闭防火墙</span></a></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#4-5-优化开机自启动服务-可选"><span class="post-toc-text">4.5 优化开机自启动服务(可选)</span></a></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#4-6-修改文件描述符"><span class="post-toc-text">4.6 修改文件描述符</span></a></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#4-7-增加集群用户sudo权限"><span class="post-toc-text">4.7 增加集群用户sudo权限</span></a></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#4-8-配置主机IP映射"><span class="post-toc-text">4.8 配置主机IP映射</span></a></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#4-9-配置SSH免密登录"><span class="post-toc-text">4.9 配置SSH免密登录</span></a></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#4-10-配置NTP时钟同步"><span class="post-toc-text">4.10 配置NTP时钟同步</span></a></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#4-11-关闭交换分区"><span class="post-toc-text">4.11 关闭交换分区</span></a></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#4-12-禁用透明大页"><span class="post-toc-text">4.12 禁用透明大页</span></a></li></ol></li><li class="post-toc-item post-toc-level-2"><a class="post-toc-link" href="#5-JDK安装"><span class="post-toc-text">5.JDK安装</span></a></li><li class="post-toc-item post-toc-level-2"><a class="post-toc-link" href="#6-配置环境变量"><span class="post-toc-text">6.配置环境变量</span></a></li><li class="post-toc-item post-toc-level-2"><a class="post-toc-link" href="#7-安装配置ZooKeeper"><span class="post-toc-text">7.安装配置ZooKeeper</span></a><ol class="post-toc-child"><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#7-1-下载ZooKeeper"><span class="post-toc-text">7.1 下载ZooKeeper</span></a></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#7-2-修改配置"><span class="post-toc-text">7.2 修改配置</span></a></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#7-3-文件分发"><span class="post-toc-text">7.3 文件分发</span></a></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#7-4-其他节点myid修改"><span class="post-toc-text">7.4 其他节点myid修改</span></a></li></ol></li><li class="post-toc-item post-toc-level-2"><a class="post-toc-link" href="#8-安装Hadoop"><span class="post-toc-text">8.安装Hadoop</span></a><ol class="post-toc-child"><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#8-1-解压"><span class="post-toc-text">8.1 解压</span></a></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#8-2-配置修改"><span class="post-toc-text">8.2 配置修改</span></a></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#8-3-创建临时文件夹和分发文件夹"><span class="post-toc-text">8.3 创建临时文件夹和分发文件夹</span></a></li></ol></li><li class="post-toc-item post-toc-level-2"><a class="post-toc-link" href="#9-启动步骤和详细过程："><span class="post-toc-text">9.启动步骤和详细过程：</span></a><ol class="post-toc-child"><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#9-1-启动ZK"><span class="post-toc-text">9.1 启动ZK</span></a></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#9-2-启动-hadoop-HDFS-YARN"><span class="post-toc-text">9.2 启动 hadoop(HDFS+YARN)</span></a><ol class="post-toc-child"><li class="post-toc-item post-toc-level-4"><a class="post-toc-link" href="#9-2-1-格式化前-先在-journalnode-节点机器上先启动-JournalNode-进程"><span class="post-toc-text">9.2.1. 格式化前,先在 journalnode 节点机器上先启动 JournalNode 进程</span></a></li><li class="post-toc-item post-toc-level-4"><a class="post-toc-link" href="#9-2-2-NameNode-格式化"><span class="post-toc-text">9.2.2.NameNode 格式化</span></a></li><li class="post-toc-item post-toc-level-4"><a class="post-toc-link" href="#9-2-3-同步-NameNode-元数据"><span class="post-toc-text">9.2.3.同步 NameNode 元数据</span></a></li><li class="post-toc-item post-toc-level-4"><a class="post-toc-link" href="#9-2-4-初始化-ZFCK"><span class="post-toc-text">9.2.4.初始化 ZFCK</span></a></li><li class="post-toc-item post-toc-level-4"><a class="post-toc-link" href="#9-2-5-启动-HDFS-分布式存储系统"><span class="post-toc-text">9.2.5.启动 HDFS 分布式存储系统</span></a></li><li class="post-toc-item post-toc-level-4"><a class="post-toc-link" href="#9-2-6-验证-namenode-datanode-zkfc"><span class="post-toc-text">9.2.6.验证 namenode,datanode,zkfc</span></a></li><li class="post-toc-item post-toc-level-4"><a class="post-toc-link" href="#9-2-7-启动-YARN-框架"><span class="post-toc-text">9.2.7.启动 YARN 框架</span></a></li><li class="post-toc-item post-toc-level-4"><a class="post-toc-link" href="#9-2-7-验证-resourcemanager-nodemanager"><span class="post-toc-text">9.2.7.验证 resourcemanager,nodemanager</span></a></li></ol></li></ol></li></ol>
        </nav>
    </aside>
    
<article id="post-Hadoop/Hadoop--Fully_Distributed _Mode_HA"
  class="post-article article-type-post fade" itemprop="blogPost">

    <div class="post-card">
        <h1 class="post-card-title">Hadoop--集群(Fully Distributed Mode)模式部署+HA</h1>
        <div class="post-meta">
            <time class="post-time" title="2017-10-03 20:33:09" datetime="2017-10-03T12:33:09.754Z"  itemprop="datePublished">2017-10-03</time>

            
	<ul class="article-category-list"><li class="article-category-list-item"><a class="article-category-list-link" href="/categories/tech/">技术</a></li></ul>



            
<span id="busuanzi_container_page_pv" title="文章总阅读量" style='display:none'>
    <i class="icon icon-eye icon-pr"></i><span id="busuanzi_value_page_pv"></span>
</span>


        </div>
        <div class="post-content" id="post-content" itemprop="postContent">
            <p>本文主要记录Hadoop全分布模式（Fully Distributed Mode）部署的详细步骤：</p>
<p>Hadoop全分布模式（Fully Distributed Mode）是一种集群部署方式，它的守护进程运行在一个集群上。</p>
<a id="more"></a>
<ul>
<li>我们一般会在master节点上部署<code>NameNode</code>,<code>ResourceManager</code>；</li>
<li><code>SecondaryNameNode</code>可以安装在master节点，也可以单独安装。</li>
<li>slave节点能看到<code>DataNode</code>和<code>NodeManager</code>。</li>
</ul>
<h2 id="1-基本环境"><a href="#1-基本环境" class="headerlink" title="1.基本环境"></a>1.基本环境</h2><ul>
<li>CentOS 6.7</li>
<li>JDK1.7+</li>
<li>Hadoop 2.8.1</li>
<li>ZooKeeper 3.4.6</li>
</ul>
<blockquote>
<p>系统原始环境特别说明：<br>系统选择minimal最小系统安装，并自定义选装以下包：<br>Base System中的Base、Compatbillty libraries、Debugging Tools; Development中的 Development tools</p>
</blockquote>
<h3 id="1-1组件说明："><a href="#1-1组件说明：" class="headerlink" title="1.1组件说明："></a>1.1组件说明：</h3><table>
<thead>
<tr>
<th>组件</th>
<th>备注说明</th>
</tr>
</thead>
<tbody>
<tr>
<td>CentOS</td>
<td>使用下列命令查看系统版本和系统位数<br>lsb_release -a #系统版本<br>file /bin/ls #系统位数</td>
</tr>
<tr>
<td>JDK</td>
<td>本文采用JDK1.8.45</td>
</tr>
<tr>
<td>Hadoop</td>
<td>使用官网最新release版本2.8.1</td>
</tr>
<tr>
<td>ZooKeeper</td>
<td>用于HA热切,存储数据使用的协调服务<br><a href="http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz" target="_blank" rel="external">下载地址：</a></td>
</tr>
</tbody>
</table>
<h2 id="2-主机节点规划"><a href="#2-主机节点规划" class="headerlink" title="2.主机节点规划"></a>2.主机节点规划</h2><table>
<thead>
<tr>
<th>HostName</th>
<th>IP Address</th>
<th>Software</th>
<th>Course</th>
</tr>
</thead>
<tbody>
<tr>
<td>hadoop001</td>
<td>192.168.194.111</td>
<td>Hadoop、ZooKeeper</td>
<td>NameNode<br>DFSZKFailoverController <br>ResourceManager<br>JobHistoryServer<br>QuorumPeerMain</td>
</tr>
<tr>
<td>hadoop002</td>
<td>192.168.194.112</td>
<td>Hadoop、ZooKeeper</td>
<td>NameNode<br>DFSZKFailoverController <br>ResourceManager<br>QuorumPeerMain</td>
</tr>
<tr>
<td>hadoop003</td>
<td>192.168.194.113</td>
<td>Hadoop、ZooKeeper</td>
<td>DataNode<br> NodeManager<br>JournalNode<br>QuorumPeerMain</td>
</tr>
<tr>
<td>hadoop004</td>
<td>192.168.194.114</td>
<td>Hadoop、ZooKeeper</td>
<td>DataNode<br> NodeManager<br>JournalNode<br>QuorumPeerMain</td>
</tr>
<tr>
<td>hadoop005</td>
<td>192.168.194.115</td>
<td>Hadoop、ZooKeeper</td>
<td>DataNode<br> NodeManager<br>JournalNode<br>QuorumPeerMain</td>
</tr>
</tbody>
</table>
<h2 id="3-目录规划"><a href="#3-目录规划" class="headerlink" title="3.目录规划"></a>3.目录规划</h2><table>
<thead>
<tr>
<th>名称</th>
<th>路径</th>
<th>备注</th>
</tr>
</thead>
<tbody>
<tr>
<td>$HADOOP_HOME</td>
<td>/opt/software/hadoop</td>
<td></td>
</tr>
<tr>
<td>Data</td>
<td>$HADOOP_HOME/data</td>
<td></td>
</tr>
<tr>
<td>Log</td>
<td>$HADOOP_HOME/logs</td>
<td></td>
</tr>
<tr>
<td>hadoop.tmp.dir</td>
<td>$HADOOP_HOME/tmp</td>
<td>需要手工创建,权限777,root:root</td>
<td></td>
</tr>
<tr>
<td>$ZOOKEEPER_HOME</td>
<td>/opt/software/zookeeper</td>
<td></td>
</tr>
</tbody>
</table>
<h2 id="4-系统环境配置"><a href="#4-系统环境配置" class="headerlink" title="4.系统环境配置"></a>4.系统环境配置</h2><p>本节内容所有节点都要配置，以下操作用hadoop001节点做示例</p>
<h3 id="4-1-配置网络和主机名"><a href="#4-1-配置网络和主机名" class="headerlink" title="4.1 配置网络和主机名"></a>4.1 配置网络和主机名</h3><h4 id="网络IP设置"><a href="#网络IP设置" class="headerlink" title="网络IP设置"></a>网络IP设置</h4><figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth0</div><div class="line"><span class="meta">#</span><span class="bash">按以下格式修改，没有的字段需要自行添加</span></div><div class="line">DEVICE=eth0                                 #网卡名称</div><div class="line">HWADDR=00:0C:29:8F:61:EB                    #MAC地址</div><div class="line">TYPE=Ethernet                               #网卡类型</div><div class="line">UUID=039afc77-da3e-4e17-bcaa-062148464812   #网卡ID</div><div class="line">ONBOOT=yes                                  #是否开机启动</div><div class="line">NM_CONTROLLED=yes                           #Network Manager</div><div class="line">BOOTPROTO=static                            #静态IP模式</div><div class="line">IPADDR=192.168.194.111                      #IP地址</div><div class="line">NETMASK=255.255.255.0                       #子网掩码</div><div class="line">GATEWAY=192.168.194.2                       #网关地址</div><div class="line">DNS1=223.5.5.5                              #DNS</div></pre></td></tr></table></figure>
<p>保存退出后，重启网卡</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 ~]# service network restart</div></pre></td></tr></table></figure>
<h4 id="主机名配置"><a href="#主机名配置" class="headerlink" title="主机名配置"></a>主机名配置</h4><p>如果在安装系统的时候未进行主机名设置，那么请根据以下命令对主机名进行配置</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 ~]# vi /etc/sysconfig/network</div><div class="line"><span class="meta">#</span><span class="bash">修改下列信息，其中HOSTNAME项修改成我们需要的主机名</span></div><div class="line">NETWORKING=yes #启动网络</div><div class="line">HOSTNAME=hadoop001 #主机名</div><div class="line"></div><div class="line"><span class="meta">#</span><span class="bash">修改完成后保存并退出，使用下列命令更新当前bash的主机名</span></div><div class="line">[root@hadoop001 ~]# hostname hadoop001</div></pre></td></tr></table></figure>
<h3 id="4-2-添加集群用户"><a href="#4-2-添加集群用户" class="headerlink" title="4.2 添加集群用户"></a>4.2 添加集群用户</h3><p>增加ID号为1000的hadoopGroup,增加ID为2000的用户并加入hadoopGroup组</p>
<p>固定用户和组的ID号是为了用户和组的权限一致，我遇到过一次在两台机器上用户名和组名相同，ID号不同结果导致权限有问题</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 ~]<span class="comment"># groupadd -g 1000 hadoopGroup</span></div><div class="line">[root@hadoop001 ~]<span class="comment"># useradd -u 2000 -g hadoopGroup hadoop</span></div></pre></td></tr></table></figure>
<p>设置hadoop用户的密码为hadoop，下面的这种方法免去了修改密码时的密码确认，在生产环境中使用此方式改密码需要清空history</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 ~]<span class="comment"># echo "hadoop"|passwd --stdin hadoop</span></div></pre></td></tr></table></figure>
<h3 id="4-3-关闭SELINUX"><a href="#4-3-关闭SELINUX" class="headerlink" title="4.3 关闭SELINUX"></a>4.3 关闭SELINUX</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 ~]<span class="comment"># sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config</span></div><div class="line"><span class="comment">#关闭当前的SELINUX状态,如果不用setenforce那么需要重启系统</span></div><div class="line">[root@hadoop001 ~]<span class="comment"># setenforce 0</span></div><div class="line">[root@hadoop001 ~]<span class="comment"># grep SELINUX=disabled /etc/selinux/config</span></div><div class="line">SELINUX=disabled</div></pre></td></tr></table></figure>
<h3 id="4-4-关闭防火墙"><a href="#4-4-关闭防火墙" class="headerlink" title="4.4 关闭防火墙"></a>4.4 关闭防火墙</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 ~]<span class="comment"># service iptables stop</span></div><div class="line">iptables: Setting chains to policy ACCEPT: filter          [  OK  ]</div><div class="line">iptables: Flushing firewall rules:                         [  OK  ]</div><div class="line">iptables: Unloading modules:                               [  OK  ]</div><div class="line">[root@hadoop001 ~]<span class="comment"># chkconfig iptables off</span></div></pre></td></tr></table></figure>
<h3 id="4-5-优化开机自启动服务-可选"><a href="#4-5-优化开机自启动服务-可选" class="headerlink" title="4.5 优化开机自启动服务(可选)"></a>4.5 优化开机自启动服务(可选)</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 ~]<span class="comment"># chkconfig --list|grep 3:on |wc -l</span></div><div class="line">[root@hadoop001 ~]<span class="comment"># chkconfig --list|grep 3:on |cut -d "" -f1</span></div><div class="line"><span class="comment">#关闭所有3级别的服务</span></div><div class="line">[root@hadoop001 ~]<span class="comment"># for name in `chkconfig --list|grep 3:on | cut -d " " -f1` ;do chkconfig $name off;done</span></div><div class="line"><span class="comment">#开启需要在3级别启动的服务，这里可以根据需要增加和修改</span></div><div class="line">[root@hadoop001 ~]<span class="comment"># for name in crond ntpd network rsyslog sshd;do chkconfig $name on;done</span></div><div class="line">[root@hadoop001 ~]<span class="comment"># chkconfig --list|grep 3:on</span></div></pre></td></tr></table></figure>
<h3 id="4-6-修改文件描述符"><a href="#4-6-修改文件描述符" class="headerlink" title="4.6 修改文件描述符"></a>4.6 修改文件描述符</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div></pre></td><td class="code"><pre><div class="line"><span class="comment">#修改系统limits.conf</span></div><div class="line">[root@hadoop001 ~]<span class="comment"># cat &gt;&gt; /etc/security/limits.conf &lt;&lt; EOF</span></div><div class="line">*           soft    nofile      262144</div><div class="line">*           hard    nofile      262144</div><div class="line">*           soft    nproc       262144</div><div class="line">*           hard    nproc       262144</div><div class="line">EOF</div><div class="line"><span class="comment">#查看系统ulimit文件句柄数</span></div><div class="line">[root@hadoop001 ~]<span class="comment"># ulimit -n</span></div><div class="line"></div><div class="line"><span class="comment">#修改90-nproc.conf</span></div><div class="line">[root@hadoop001 ~]<span class="comment"># vi /etc/security/limits.d/90-nproc.conf</span></div><div class="line">*  soft nproc 262144</div></pre></td></tr></table></figure>
<h3 id="4-7-增加集群用户sudo权限"><a href="#4-7-增加集群用户sudo权限" class="headerlink" title="4.7 增加集群用户sudo权限"></a>4.7 增加集群用户sudo权限</h3><p>增加hadoop用户的sudo权限</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 ~]<span class="comment"># vi /etc/sudoers</span></div><div class="line">hadoop ALL=(ALL)       ALL</div></pre></td></tr></table></figure>
<h3 id="4-8-配置主机IP映射"><a href="#4-8-配置主机IP映射" class="headerlink" title="4.8 配置主机IP映射"></a>4.8 配置主机IP映射</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 ~]<span class="comment"># vi /etc/hosts</span></div><div class="line"><span class="comment">#在hosts文件末尾追加</span></div><div class="line"></div><div class="line">192.168.194.111    hadoop001</div><div class="line">192.168.194.112    hadoop002</div><div class="line">192.168.194.113    hadoop003</div><div class="line">192.168.194.114    hadoop004</div><div class="line">192.168.194.115    hadoop005</div></pre></td></tr></table></figure>
<h3 id="4-9-配置SSH免密登录"><a href="#4-9-配置SSH免密登录" class="headerlink" title="4.9 配置SSH免密登录"></a>4.9 配置SSH免密登录</h3><p>SSH免密码登录配置的详细配置步骤：<a href="http://blog.xiejm.com/41.html" target="_blank" rel="external">SSH免密登录</a></p>
<h3 id="4-10-配置NTP时钟同步"><a href="#4-10-配置NTP时钟同步" class="headerlink" title="4.10 配置NTP时钟同步"></a>4.10 配置NTP时钟同步</h3><ul>
<li>第一种方法 ：（所有节点都要操作）都从公共NTP服务器同步，执行如下：</li>
</ul>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">$ cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime</div><div class="line">$ ntpdate us.pool.ntp.org</div><div class="line">$ crontab -e</div><div class="line">0-59/10 * * * * /usr/sbin/ntpdate us.pool.ntp.org | logger -t NTP</div></pre></td></tr></table></figure>
<p>以上crontab的配置说明详见 <a href="http://note.youdao.com/" target="_blank" rel="external">Linux-crontab命令</a></p>
<ul>
<li>第二种方法：选一个节点搭建一个NTP服务，其他节点从该NTP服务器同步，<a href="http://note.youdao.com/" target="_blank" rel="external">具体步骤参考本博客的NTP 配置</a></li>
</ul>
<h3 id="4-11-关闭交换分区"><a href="#4-11-关闭交换分区" class="headerlink" title="4.11 关闭交换分区"></a>4.11 关闭交换分区</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 ~]<span class="comment"># echo '0' &gt; /proc/sys/vm/swappiness</span></div><div class="line">[root@hadoop001 ~]<span class="comment"># sysctl vm.swappiness=0</span></div><div class="line">[root@hadoop001 ~]<span class="comment"># echo 'vm.swappiness=0'&gt;&gt; /etc/sysctl.conf</span></div></pre></td></tr></table></figure>
<h3 id="4-12-禁用透明大页"><a href="#4-12-禁用透明大页" class="headerlink" title="4.12 禁用透明大页"></a>4.12 禁用透明大页</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 ~]<span class="comment"># echo never &gt; /sys/kernel/mm/redhat_transparent_hugepage/defrag</span></div><div class="line">[root@hadoop001 ~]<span class="comment"># cat /sys/kernel/mm/redhat_transparent_hugepage/defrag</span></div><div class="line">always [never]</div><div class="line">[root@hadoop001 ~]<span class="comment"># echo 'echo never &gt; /sys/kernel/mm/redhat_transparent_hugepage/defrag ' &gt;&gt; /etc/rc.local</span></div></pre></td></tr></table></figure>
<h2 id="5-JDK安装"><a href="#5-JDK安装" class="headerlink" title="5.JDK安装"></a>5.JDK安装</h2><p><strong>【注意】所有节点配置</strong><br>详细安装步骤参考本博客的JDK安装的文章:<a href="http://blog.xiejm.com/8.html" target="_blank" rel="external">JDK安装步骤</a></p>
<h2 id="6-配置环境变量"><a href="#6-配置环境变量" class="headerlink" title="6.配置环境变量"></a>6.配置环境变量</h2><ul>
<li><strong>【注意】所有节点配置</strong></li>
</ul>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 software]<span class="comment"># vim /etc/profile</span></div><div class="line"></div><div class="line"><span class="comment">#按组合键shift+g 或大写G 到最后一行，按字母o  插入下列内容</span></div><div class="line">    <span class="comment">#SET HADOOP</span></div><div class="line">    exprot HADOOP_HOME=/opt/software/hadoop</div><div class="line"></div><div class="line">    <span class="comment">#SET ZOOKEEPER</span></div><div class="line">    <span class="built_in">export</span> ZOOKEEPER_HOME=/opt/software/zookeeper</div><div class="line"><span class="comment">#如果JDK环境变量没有配置，请加入下列内容</span></div><div class="line">    <span class="comment">#SET JDK</span></div><div class="line">    <span class="built_in">export</span> JAVA_HOME=/usr/java/jdk1.7.0_79</div><div class="line">    <span class="built_in">export</span> JRE_HOME=<span class="variable">$JAVA_HOME</span>/jre</div><div class="line">    <span class="built_in">export</span> CLASS_PATH=.:<span class="variable">$JAVA_HOME</span>/lib:<span class="variable">$JAVA_HOME</span>/lib/tools.jar</div><div class="line"></div><div class="line"><span class="comment">#设置PATH</span></div><div class="line">    <span class="built_in">export</span> PATH=<span class="variable">$ZOOKEEPER_HOME</span>/bin:<span class="variable">$HADOOP_HOME</span>/bin:<span class="variable">$HADOOP_HOME</span>/sbin:<span class="variable">$JAVA_HOME</span>/bin:<span class="variable">$JRE_HOME</span>/bin:<span class="variable">$PATH</span></div><div class="line"></div><div class="line"><span class="comment">#添加完成后  :wq 保存</span></div><div class="line"><span class="comment">#使修改生效：</span></div><div class="line">[root@hadoop001 software]<span class="comment">#source !$</span></div><div class="line"></div><div class="line"><span class="comment">#验证JDK有效性</span></div><div class="line">[root@hadoop001 software]<span class="comment"># java -version</span></div><div class="line">java version <span class="string">"1.7.0_79"</span></div><div class="line">Java(TM) SE Runtime Environment (build 1.7.0_79-b15)</div><div class="line">Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)</div></pre></td></tr></table></figure>
<h2 id="7-安装配置ZooKeeper"><a href="#7-安装配置ZooKeeper" class="headerlink" title="7.安装配置ZooKeeper"></a>7.安装配置ZooKeeper</h2><h3 id="7-1-下载ZooKeeper"><a href="#7-1-下载ZooKeeper" class="headerlink" title="7.1 下载ZooKeeper"></a>7.1 下载ZooKeeper</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 software]<span class="comment"># pwd</span></div><div class="line">/opt/software</div><div class="line">[root@hadoop001 software]<span class="comment"># wget http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz</span></div><div class="line">[root@hadoop001 software]<span class="comment"># tar -zxvf zookeeper-3.4.6.tar.gz</span></div><div class="line">[root@hadoop001 software]<span class="comment"># mv zookeeper-3.4.6 zookeeper</span></div><div class="line">[root@hadoop001 software]<span class="comment"># chown -R hadoop:hadoopGroup zookeeper</span></div></pre></td></tr></table></figure>
<h3 id="7-2-修改配置"><a href="#7-2-修改配置" class="headerlink" title="7.2 修改配置"></a>7.2 修改配置</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 software]<span class="comment"># cd /opt/software/zookeeper/conf</span></div><div class="line">[root@hadoop001 conf]<span class="comment"># cp zoo_sample.cfg zoo.cfg</span></div><div class="line"></div><div class="line">[root@hadoop001 conf]<span class="comment"># vi zoo.cfg</span></div><div class="line">修改dataDir</div><div class="line">dataDir=/opt/software/zookeeper/data</div><div class="line">添加下面三行</div><div class="line">server.1=hadoop001:2888:3888</div><div class="line">server.2=hadoop002:2888:3888</div><div class="line">server.3=hadoop003:2888:3888</div><div class="line">server.2=hadoop004:2888:3888</div><div class="line">server.3=hadoop005:2888:3888</div><div class="line"></div><div class="line">[root@hadoop001 conf]<span class="comment"># cd ../</span></div><div class="line">[root@hadoop001 zookeeper]<span class="comment"># mkdir data</span></div><div class="line">[root@hadoop001 zookeeper]<span class="comment"># touch data/myid</span></div><div class="line">[root@hadoop001 zookeeper]<span class="comment"># echo 1 &gt; data/myid</span></div><div class="line">[root@hadoop001 zookeeper]<span class="comment"># cat data/myid</span></div><div class="line">1</div></pre></td></tr></table></figure>
<h3 id="7-3-文件分发"><a href="#7-3-文件分发" class="headerlink" title="7.3 文件分发"></a>7.3 文件分发</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 zookeeper]<span class="comment"># scp -r /opt/software/zookeeper root@hadoop002:/opt/software</span></div><div class="line">[root@hadoop001 zookeeper]<span class="comment"># scp -r /opt/software/zookeeper root@hadoop003:/opt/software</span></div><div class="line">[root@hadoop001 zookeeper]<span class="comment"># scp -r /opt/software/zookeeper root@hadoop004:/opt/software</span></div><div class="line">[root@hadoop001 zookeeper]<span class="comment"># scp -r /opt/software/zookeeper root@hadoop005:/opt/software</span></div></pre></td></tr></table></figure>
<h3 id="7-4-其他节点myid修改"><a href="#7-4-其他节点myid修改" class="headerlink" title="7.4 其他节点myid修改"></a>7.4 其他节点myid修改</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line"><span class="comment">#不同zk节点配置不同的myid</span></div><div class="line"><span class="comment">#myid中的数值需要和 zoo.cfg中的配置一致。</span></div><div class="line"><span class="comment"># hadoop002/003/004/005,也修改配置,就如下不同</span></div><div class="line">[root@hadoop002 zookeeper]<span class="comment"># echo 2 &gt; data/myid</span></div><div class="line">[root@hadoop003 zookeeper]<span class="comment"># echo 3 &gt; data/myid</span></div><div class="line">[root@hadoop004 zookeeper]<span class="comment"># echo 4 &gt; data/myid</span></div><div class="line">[root@hadoop005 zookeeper]<span class="comment"># echo 5 &gt; data/myid</span></div><div class="line"><span class="comment">####注意！！！不能把 echo 3&gt;data/myid,将&gt;前后空格保留,否则无法将 3 写入myid文件</span></div></pre></td></tr></table></figure>
<h2 id="8-安装Hadoop"><a href="#8-安装Hadoop" class="headerlink" title="8.安装Hadoop"></a>8.安装Hadoop</h2><h3 id="8-1-解压"><a href="#8-1-解压" class="headerlink" title="8.1 解压"></a>8.1 解压</h3><p>之前编译好了hadoop-2.8.1-snappy.tar.gz，上传这个包并解压。<br>具体编译过程，参考本博客<a href="http://note.youdao.com/" target="_blank" rel="external"></a></p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 ~]<span class="comment"># mkdir -p /opt/software</span></div><div class="line"><span class="comment">#将hadoop安装包放到/opt/software</span></div><div class="line">[root@hadoop001 ~]<span class="comment"># cd /opt/software</span></div><div class="line">[root@hadoop001 software]<span class="comment"># tar -zxvf hadoop-2.8.1.tar.gz</span></div><div class="line"></div><div class="line"><span class="comment">#设置软连接</span></div><div class="line">[root@hadoop001 software]<span class="comment"># ln -s /opt/software/hadoop-2.8.1 /opt/software/hadoop</span></div><div class="line"></div><div class="line"><span class="comment"># 当我们使用hadoop用户才需要下面命令</span></div><div class="line"><span class="comment">#设置hadoop目录所有者、所属组</span></div><div class="line"><span class="comment">#[root@hadoop001 software]# chown -R hadoop:hadoopGroup hadoop/*</span></div><div class="line"><span class="comment">#[root@hadoop001 software]# chown -R hadoop:hadoopGroup hadoop-2.8.1</span></div><div class="line"><span class="comment">#[root@hadoop001 software]# chown -R hadoop:hadoopGroup hadoop-2.8.1/*</span></div></pre></td></tr></table></figure>
<h3 id="8-2-配置修改"><a href="#8-2-配置修改" class="headerlink" title="8.2 配置修改"></a>8.2 配置修改</h3><p>使用命令<code>cd $HADOOP_HOME/etc/hadoop</code>进入配置文件目录</p>
<ul>
<li>8.2.1 修改$HADOOP_HOME/etc/hadoop/hadoop-env.sh</li>
</ul>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="built_in">export</span> JAVA_HOME=<span class="string">"/usr/java/jdk1.7.0_67"</span></div><div class="line"><span class="built_in">export</span> HADOOP_OPTS=<span class="string">"<span class="variable">$HADOOP_OPTS</span> -Djava.library.path=<span class="variable">$HADOOP_HOME</span>/lib:<span class="variable">$HADOOP_HOME</span>/lib/native"</span></div></pre></td></tr></table></figure>
<ul>
<li><p>8.2.2 修改$HADOOP_HOME/etc/hadoop/core-site.xml</p>
</li>
<li><p>8.2.3 修改$HADOOP_HOME/etc/hadoop/hdfs-site.xml</p>
</li>
<li><p>8.2.4 修改$HADOOP_HOEM/etc/hadoop/mapred-site.xml</p>
<p>  <em>首先复制mapred-site.xml的模板文件</em>，<br>然后修改mapred-site.xml</p>
</li>
</ul>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">[root@sht-sgmhadoopnn-01 hadoop]<span class="comment"># cp  mapred-site.xml.template  mapred-site.xml</span></div><div class="line">[root@sht-sgmhadoopnn-01 hadoop]<span class="comment"># vi  mapred-site.xml</span></div></pre></td></tr></table></figure>
<ul>
<li><p>8.2.5 修改$HADOOP_HOME/etc/hadoop/yarn-site.xml</p>
</li>
<li><p>8.2.6 修改slaves</p>
</li>
</ul>
<p><a href="http://wpxiejm.oss-cn-beijing.aliyuncs.com/Note/hadoop/Hadoop_HA_conf.rar" target="_blank" rel="external">点击下载配置文件</a>，解压后复制到$HADOOP_HOME/etc/hadoop目录下</p>
<h3 id="8-3-创建临时文件夹和分发文件夹"><a href="#8-3-创建临时文件夹和分发文件夹" class="headerlink" title="8.3 创建临时文件夹和分发文件夹"></a>8.3 创建临时文件夹和分发文件夹</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 hadoop]<span class="comment"># mkdir -p  /opt/software/hadoop/tmp</span></div><div class="line">[root@hadoop001 hadoop]<span class="comment"># chmod -R 777  /opt/software/hadoop/tmp</span></div><div class="line">[root@hadoop001 hadoop]<span class="comment"># chown -R root:root  /opt/software/hadoop/tmp</span></div><div class="line"></div><div class="line">[root@hadoop001 hadoop]<span class="comment"># scp -r  /opt/software/hadoop root@hadoop002:/hadoop</span></div><div class="line">[root@hadoop001 hadoop]<span class="comment"># scp -r  /opt/software/hadoop root@hadoop003:/hadoop</span></div><div class="line">[root@hadoop001 hadoop]<span class="comment"># scp -r  /opt/software/hadoop root@hadoop004:/hadoop</span></div><div class="line">[root@hadoop001 hadoop]<span class="comment"># scp -r  /opt/software/hadoop root@hadoop005:/hadoop</span></div></pre></td></tr></table></figure>
<h2 id="9-启动步骤和详细过程："><a href="#9-启动步骤和详细过程：" class="headerlink" title="9.启动步骤和详细过程："></a>9.启动步骤和详细过程：</h2><h3 id="9-1-启动ZK"><a href="#9-1-启动ZK" class="headerlink" title="9.1 启动ZK"></a>9.1 启动ZK</h3><p>在所有的ZK节点执行命令： <code>zkServer.sh start</code></p>
<p>可借助命令 <code>zkServer.sh status</code>  查看各个ZK的从属关系</p>
<h3 id="9-2-启动-hadoop-HDFS-YARN"><a href="#9-2-启动-hadoop-HDFS-YARN" class="headerlink" title="9.2 启动 hadoop(HDFS+YARN)"></a>9.2 启动 hadoop(HDFS+YARN)</h3><h4 id="9-2-1-格式化前-先在-journalnode-节点机器上先启动-JournalNode-进程"><a href="#9-2-1-格式化前-先在-journalnode-节点机器上先启动-JournalNode-进程" class="headerlink" title="9.2.1. 格式化前,先在 journalnode 节点机器上先启动 JournalNode 进程"></a>9.2.1. 格式化前,先在 journalnode 节点机器上先启动 JournalNode 进程</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div></pre></td><td class="code"><pre><div class="line">[root@hadoop003 ~]<span class="comment"># cd /opt/software/hadoop/sbin</span></div><div class="line">[root@hadoop003 sbin]<span class="comment"># hadoop-daemon.sh start journalnode</span></div><div class="line">starting journalnode, logging to /opt/software/hadoop/logs/hadoop-root-journalnode-hadoop001.out</div><div class="line">[root@hadoop003 sbin]<span class="comment"># jps</span></div><div class="line">4016 Jps</div><div class="line">3683 QuorumPeerMain</div><div class="line">3981 JournalNode</div><div class="line"></div><div class="line"></div><div class="line">[root@hadoop004 hadoop]<span class="comment"># cd /opt/software/hadoop/sbin</span></div><div class="line">[root@hadoop004 sbin]<span class="comment"># hadoop-daemon.sh start journalnode</span></div><div class="line">starting journalnode, logging to /opt/software/hadoop/logs/hadoop-root-journalnode-hadoop002.out</div><div class="line">[root@hadoop004 sbin]<span class="comment"># jps</span></div><div class="line">9891 Jps</div><div class="line">9609 QuorumPeerMain</div><div class="line">9852 JournalNode</div><div class="line"></div><div class="line"></div><div class="line">[root@hadoop005 hadoop]<span class="comment"># cd /opt/software/hadoop/sbin</span></div><div class="line">[root@hadoop005 sbin]<span class="comment"># hadoop-daemon.sh start journalnode</span></div><div class="line">starting journalnode, logging to /opt/software/hadoop/logs/hadoop-root-journalnode-hadoop003.out</div><div class="line">[root@hadoop005 sbin]<span class="comment"># jps</span></div><div class="line">4425 JournalNode</div><div class="line">4460 Jps</div><div class="line">4191 QuorumPeerMain</div></pre></td></tr></table></figure>
<h4 id="9-2-2-NameNode-格式化"><a href="#9-2-2-NameNode-格式化" class="headerlink" title="9.2.2.NameNode 格式化"></a>9.2.2.NameNode 格式化</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 sbin]<span class="comment"># cd ../</span></div><div class="line">[root@hadoop001 hadoop]<span class="comment"># hadoop namenode -format</span></div></pre></td></tr></table></figure>
<h4 id="9-2-3-同步-NameNode-元数据"><a href="#9-2-3-同步-NameNode-元数据" class="headerlink" title="9.2.3.同步 NameNode 元数据"></a>9.2.3.同步 NameNode 元数据</h4><p>同步 hadoop001 元数据到 hadoop002<br>主要是： dfs.namenode.name.dir， dfs.namenode.edits.dir 还应该确保共享存储目录下<br>(dfs.namenode.shared.edits.dir ) 包含 NameNode 所有的元数据。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 hadoop]<span class="comment"># pwd</span></div><div class="line">/opt/software/hadoop</div><div class="line">[root@hadoop001 hadoop]<span class="comment"># scp -r data/ root@hadoop002:/opt/software/hadoop</span></div><div class="line">in_use.lock 100% 14 0.0KB/s 00:00</div><div class="line">VERSION 100% 167 0.2KB/s 00:00</div><div class="line">seen_txid 100% 2 0.0KB/s 00:00</div><div class="line">VERSION 100% 220 0.2KB/s 00:00</div><div class="line">fsimage_0000000000000000000.md5 100% 62 0.1KB/s</div><div class="line">00:00</div><div class="line">fsimage_0000000000000000000 100% 306 0.3KB/s</div><div class="line">00:00</div><div class="line">[root@hadoop001 hadoop]<span class="comment">#</span></div></pre></td></tr></table></figure>
<h4 id="9-2-4-初始化-ZFCK"><a href="#9-2-4-初始化-ZFCK" class="headerlink" title="9.2.4.初始化 ZFCK"></a>9.2.4.初始化 ZFCK</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 bin]<span class="comment"># hdfs zkfc -formatZK</span></div></pre></td></tr></table></figure>
<h4 id="9-2-5-启动-HDFS-分布式存储系统"><a href="#9-2-5-启动-HDFS-分布式存储系统" class="headerlink" title="9.2.5.启动 HDFS 分布式存储系统"></a>9.2.5.启动 HDFS 分布式存储系统</h4><p>集群启动,在 hadoop001 执行 start-dfs.sh<br>集群关闭,在 hadoop001 执行 stop-dfs.sh</p>
<ul>
<li><strong>集群启动</strong></li>
</ul>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 sbin]<span class="comment"># start-dfs.sh</span></div><div class="line">[root@hadoop001 hadoop]<span class="comment"># start-dfs.sh</span></div><div class="line"><span class="comment">#补充集群启动信息</span></div></pre></td></tr></table></figure>
<ul>
<li><strong>如果需要使用单独启动某个进程，请使用以下命令</strong></li>
</ul>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line">NameNode(hadoop001, hadoop002):</div><div class="line">hadoop-daemon.sh start namenode</div><div class="line">DataNode(hadoop003, hadoop004, hadoop005):</div><div class="line">hadoop-daemon.sh start datanode</div><div class="line">JournamNode(hadoop003, hadoop003, hadoop005):</div><div class="line">hadoop-daemon.sh start journalnode</div><div class="line">ZKFC(hadoop001, hadoop002):</div><div class="line">hadoop-daemon.sh start zkfc</div></pre></td></tr></table></figure>
<h4 id="9-2-6-验证-namenode-datanode-zkfc"><a href="#9-2-6-验证-namenode-datanode-zkfc" class="headerlink" title="9.2.6.验证 namenode,datanode,zkfc"></a>9.2.6.验证 namenode,datanode,zkfc</h4><ul>
<li>HDFS进程检查</li>
</ul>
<p><code>hadoop001</code>和<code>hadoop002</code>节点的进程相同，<code>hadoop003</code>、<code>hadoop004</code>、<code>hadoop005</code>节点的进程相同,下面用hadoop001、hadoop003的jps 命令输出为示例</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 sbin]<span class="comment"># jps</span></div><div class="line">12712 Jps</div><div class="line">12593 DFSZKFailoverController</div><div class="line">12278 NameNode</div><div class="line"></div><div class="line">[root@hadoop003 ~]<span class="comment"># jps</span></div><div class="line">6348 JournalNode</div><div class="line">8775 Jps</div><div class="line">559 QuorumPeerMain</div><div class="line">8509 DataNode</div></pre></td></tr></table></figure>
<ul>
<li>webUI页面检查NN、DN节点情况</li>
</ul>
<p>hadoop001:<br><a href="http://hadoop001:50070/" target="_blank" rel="external">http://hadoop001:50070/</a></p>
<p>hadoop002:<br><a href="http://hadoop002:50070/" target="_blank" rel="external">http://hadoop002:50070/</a></p>
<h4 id="9-2-7-启动-YARN-框架"><a href="#9-2-7-启动-YARN-框架" class="headerlink" title="9.2.7.启动 YARN 框架"></a>9.2.7.启动 YARN 框架</h4><ul>
<li><strong>集群启动</strong><br>在hadoop001 启动 Yarn，命令所在目录： $HADOOP_HOME/sbin</li>
</ul>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 hadoop]<span class="comment"># start-yarn.sh</span></div><div class="line">starting yarn daemons</div><div class="line">starting resourcemanager, logging to /opt/software/hadoop/logs/yarn-root-resourcemanagerhadoop001.out</div><div class="line">hadoop002: starting nodemanager, logging to /opt/software/hadoop/logs/yarn-rootnodemanager-hadoop002.out</div><div class="line">hadoop003: starting nodemanager, logging to /opt/software/hadoop/logs/yarn-rootnodemanager-hadoop003.out</div><div class="line">hadoop001: starting nodemanager, logging to /opt/software/hadoop/logs/yarn-rootnodemanager-hadoop001.out</div><div class="line">[root@hadoop001 hadoop]<span class="comment">#</span></div></pre></td></tr></table></figure>
<ul>
<li>在hadoop002 备机启动 RM</li>
</ul>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">[root@hadoop002 ~]<span class="comment"># yarn-daemon.sh start resourcemanager</span></div><div class="line">starting resourcemanager, logging to /opt/software/hadoop/logs/yarn-root-resourcemanagerhadoop002.out</div><div class="line">[root@hadoop002 ~]<span class="comment">#</span></div></pre></td></tr></table></figure>
<ul>
<li><strong>如果需要单独启动，请使用下面命令</strong></li>
</ul>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line">1) ResourceManager(hadoop001, hadoop002)</div><div class="line">yarn-daemon.sh start resourcemanager</div><div class="line">2) NodeManager(hadoop001, hadoop002, hadoop003)</div><div class="line">yarn-daemon.sh start nodemanager</div><div class="line"></div><div class="line"></div><div class="line"></div><div class="line">- **如果需要关闭YARN集群可以使用下面命令**</div><div class="line">[root@hadoop001 sbin]<span class="comment"># stop-yarn.sh</span></div><div class="line"><span class="comment">#包含 namenode 的 resourcemanager 进程， datanode 的 nodemanager 进程</span></div><div class="line">[root@hadoop002 sbin]<span class="comment"># yarn-daemon.sh stop resourcemanager</span></div></pre></td></tr></table></figure>
<h4 id="9-2-7-验证-resourcemanager-nodemanager"><a href="#9-2-7-验证-resourcemanager-nodemanager" class="headerlink" title="9.2.7.验证 resourcemanager,nodemanager"></a>9.2.7.验证 resourcemanager,nodemanager</h4><ul>
<li>YARN进程验证</li>
</ul>
<p><code>hadoop001</code>和<code>hadoop002</code>节点的进程相同，<code>hadoop003</code>、<code>hadoop004</code>、<code>hadoop005</code>节点的进程相同,下面用hadoop001、hadoop003的jps 命令输出为示例</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div></pre></td><td class="code"><pre><div class="line">[root@hadoop001 ~]<span class="comment"># jps</span></div><div class="line">5043 NodeManager</div><div class="line">3683 QuorumPeerMain</div><div class="line">4679 DFSZKFailoverController</div><div class="line">5355 Jps</div><div class="line">4348 DataNode</div><div class="line">3981 JournalNode</div><div class="line">4943 ResourceManager</div><div class="line"></div><div class="line"></div><div class="line">[root@hadoop003 ~]<span class="comment"># jps</span></div><div class="line">4518 DataNode</div><div class="line">4679 NodeManager</div><div class="line">4425 JournalNode</div><div class="line">4794 Jps</div><div class="line">4191 QuorumPeerMain</div></pre></td></tr></table></figure>
<ul>
<li>页面验证</li>
</ul>
<p>ResourceManger（ Active） ： <a href="http://192.168.194.111:8088" target="_blank" rel="external">http://192.168.194.111:8088</a></p>
<p>ResourceManger（ Standby）： <a href="http://192.168.194.112:8088/cluster/cluster" target="_blank" rel="external">http://192.168.194.112:8088/cluster/cluster</a></p>

        </div>

        <blockquote class="post-copyright">
    <div class="content">
        
<span class="post-time">
    最后更新时间：<time datetime="2017-10-03T13:06:39.437Z" itemprop="dateUpdated">2017-10-03 21:06:39</time>
</span><br>


        
        原始链接：<a href="/Hadoop/Hadoop--Fully_Distributed _Mode_HA.html" target="_blank" rel="external">http://xiejm.com/Hadoop/Hadoop--Fully_Distributed _Mode_HA.html</a>
        
    </div>
    <footer>
        <a href="http://xiejm.com">
            <img src="/img/avatar.jpg" alt="XieJM">
            XieJM
        </a>
    </footer>
</blockquote>

        


        <div class="post-footer">
            
	<ul class="article-tag-list"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/tags/Hadoop/">Hadoop</a></li></ul>


            
<div class="page-share-wrap">
    

<div class="page-share" id="pageShare">
    <ul class="reset share-icons">
      <li>
        <a class="weibo share-sns" target="_blank" href="http://service.weibo.com/share/share.php?url=http://xiejm.com/Hadoop/Hadoop--Fully_Distributed _Mode_HA.html&title=《Hadoop--集群(Fully Distributed Mode)模式部署+HA》 — XieJM's Blog&pic=http://xiejm.com/img/avatar.jpg" data-title="微博">
          <i class="icon icon-weibo"></i>
        </a>
      </li>
      <li>
        <a class="weixin share-sns wxFab" href="javascript:;" data-title="微信">
          <i class="icon icon-weixin"></i>
        </a>
      </li>
      <li>
        <a class="qq share-sns" target="_blank" href="http://connect.qq.com/widget/shareqq/index.html?url=http://xiejm.com/Hadoop/Hadoop--Fully_Distributed _Mode_HA.html&title=《Hadoop--集群(Fully Distributed Mode)模式部署+HA》 — XieJM's Blog&source=本文主要记录Hadoop全分布模式（Fully Distributed Mode）部署的详细步骤：
Hadoop全分布模式（Fully Distribut..." data-title=" QQ">
          <i class="icon icon-qq"></i>
        </a>
      </li>
      <li>
        <a class="facebook share-sns" target="_blank" href="https://www.facebook.com/sharer/sharer.php?u=http://xiejm.com/Hadoop/Hadoop--Fully_Distributed _Mode_HA.html" data-title=" Facebook">
          <i class="icon icon-facebook"></i>
        </a>
      </li>
      <li>
        <a class="twitter share-sns" target="_blank" href="https://twitter.com/intent/tweet?text=《Hadoop--集群(Fully Distributed Mode)模式部署+HA》 — XieJM's Blog&url=http://xiejm.com/Hadoop/Hadoop--Fully_Distributed _Mode_HA.html&via=http://xiejm.com" data-title=" Twitter">
          <i class="icon icon-twitter"></i>
        </a>
      </li>
      <li>
        <a class="google share-sns" target="_blank" href="https://plus.google.com/share?url=http://xiejm.com/Hadoop/Hadoop--Fully_Distributed _Mode_HA.html" data-title=" Google+">
          <i class="icon icon-google-plus"></i>
        </a>
      </li>
    </ul>
 </div>



    <a href="javascript:;" id="shareFab" class="page-share-fab waves-effect waves-circle">
        <i class="icon icon-share-alt icon-lg"></i>
    </a>
</div>



        </div>
    </div>

    
<nav class="post-nav flex-row flex-justify-between">
  
    <div class="waves-block waves-effect prev">
      <a href="/Hive/Hive--load_data.html" id="post-prev" class="post-nav-link">
        <div class="tips"><i class="icon icon-angle-left icon-lg icon-pr"></i> Prev</div>
        <h4 class="title">Hive的几种数据导入方式</h4>
      </a>
    </div>
  

  
    <div class="waves-block waves-effect next">
      <a href="/Hadoop/Hadoop--HDFS_Shell.html" id="post-next" class="post-nav-link">
        <div class="tips">Next <i class="icon icon-angle-right icon-lg icon-pl"></i></div>
        <h4 class="title">Hadoop--HDFS的shell(命令行客户端)操作</h4>
      </a>
    </div>
  
</nav>



    


<section class="comments" id="comments">
    <div id="disqus_thread"></div>
    <script>
    var disqus_shortname = 'true';
    lazyScripts.push('//' + disqus_shortname + '.disqus.com/embed.js')
    </script>
    <noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>
</section>













</article>



</div>

    </main>
    <div class="mask" id="mask"></div>
<a href="javascript:;" id="gotop" class="waves-effect waves-circle waves-light"><span class="icon icon-lg icon-chevron-up"></span></a>



<div class="global-share" id="globalShare">
    <ul class="reset share-icons">
      <li>
        <a class="weibo share-sns" target="_blank" href="http://service.weibo.com/share/share.php?url=http://xiejm.com/Hadoop/Hadoop--Fully_Distributed _Mode_HA.html&title=《Hadoop--集群(Fully Distributed Mode)模式部署+HA》 — XieJM's Blog&pic=http://xiejm.com/img/avatar.jpg" data-title="微博">
          <i class="icon icon-weibo"></i>
        </a>
      </li>
      <li>
        <a class="weixin share-sns wxFab" href="javascript:;" data-title="微信">
          <i class="icon icon-weixin"></i>
        </a>
      </li>
      <li>
        <a class="qq share-sns" target="_blank" href="http://connect.qq.com/widget/shareqq/index.html?url=http://xiejm.com/Hadoop/Hadoop--Fully_Distributed _Mode_HA.html&title=《Hadoop--集群(Fully Distributed Mode)模式部署+HA》 — XieJM's Blog&source=本文主要记录Hadoop全分布模式（Fully Distributed Mode）部署的详细步骤：
Hadoop全分布模式（Fully Distribut..." data-title=" QQ">
          <i class="icon icon-qq"></i>
        </a>
      </li>
      <li>
        <a class="facebook share-sns" target="_blank" href="https://www.facebook.com/sharer/sharer.php?u=http://xiejm.com/Hadoop/Hadoop--Fully_Distributed _Mode_HA.html" data-title=" Facebook">
          <i class="icon icon-facebook"></i>
        </a>
      </li>
      <li>
        <a class="twitter share-sns" target="_blank" href="https://twitter.com/intent/tweet?text=《Hadoop--集群(Fully Distributed Mode)模式部署+HA》 — XieJM's Blog&url=http://xiejm.com/Hadoop/Hadoop--Fully_Distributed _Mode_HA.html&via=http://xiejm.com" data-title=" Twitter">
          <i class="icon icon-twitter"></i>
        </a>
      </li>
      <li>
        <a class="google share-sns" target="_blank" href="https://plus.google.com/share?url=http://xiejm.com/Hadoop/Hadoop--Fully_Distributed _Mode_HA.html" data-title=" Google+">
          <i class="icon icon-google-plus"></i>
        </a>
      </li>
    </ul>
 </div>


<div class="page-modal wx-share" id="wxShare">
    <a class="close" href="javascript:;"><i class="icon icon-close"></i></a>
    <p>扫一扫，分享到微信</p>
    <img src="" alt="微信分享二维码">
</div>




    <script src="//cdn.bootcss.com/node-waves/0.7.4/waves.min.js"></script>
<script>
var BLOG = { ROOT: '/', SHARE: true, REWARD: false };


</script>

<script src="/js/main.min.js?v=1.6.13"></script>


<div class="search-panel" id="search-panel">
    <ul class="search-result" id="search-result"></ul>
</div>
<template id="search-tpl">
<li class="item">
    <a href="{path}" class="waves-block waves-effect">
        <div class="title ellipsis" title="{title}">{title}</div>
        <div class="flex-row flex-middle">
            <div class="tags ellipsis">
                {tags}
            </div>
            <time class="flex-col time">{date}</time>
        </div>
    </a>
</li>
</template>

<script src="/js/search.min.js?v=1.6.13" async></script>






<script async src="//dn-lbstatics.qbox.me/busuanzi/2.3/busuanzi.pure.mini.js"></script>



<script>
(function() {
    var OriginTitile = document.title, titleTime;
    document.addEventListener('visibilitychange', function() {
        if (document.hidden) {
            document.title = 'XieJM's Blog';
            clearTimeout(titleTime);
        } else {
            document.title = 'XieJM's Blog';
            titleTime = setTimeout(function() {
                document.title = OriginTitile;
            },2000);
        }
    });
})();
</script>



</body>
</html>
